Papers by Roy Ka-Wei Lee
Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment (2026.acl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Zhengyuan Liu, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Roy Ka-Wei Lee
| Challenge: | Current alignment paradigms treat "human values" as a monolithic entity, ignoring the fact that many societies are a mosaic of diverse subgroups with distinct and sometimes conflicting values, preferences, and norms. |
| Approach: | They examine whether Large Language Models can emulate distinct cultural values of subgroups . they use a global value survey to examine the value landscape of a multicultural society . |
| Outcome: | The proposed model improves on unseen, out-of-distribution subgroups by 17.4% . the model widens the disparity between subgroup groups when measured by distance-aware metrics. |
Improving the Detection of Multilingual Online Attacks with Rich Social Media Data from Singapore (2023.acl-long)
Copied to clipboard
Janosch Haber, Bertie Vidgen, Matthew Chapman, Vibhor Agarwal, Roy Ka-Wei Lee, Yong Keong Yap, Paul Röttger
| Challenge: | Toxic content is a global problem, but most resources for detecting toxic content are in English . new datasets and models for non-English languages focus exclusively on one language or dialect . |
| Approach: | They propose to use a multilingual dataset of online attacks to identify code-mixed toxic content in Singapore . they collect reddit comments in Indonesian, Malay, Singlish, and other languages and provide fine-grained hierarchical labels for attacks . |
| Outcome: | The proposed dataset provides fine-grained hierarchical labels for online attacks in Singapore . it shows that the metadata can be used for granular error analysis . |
Understanding Refusal in Language Models with Sparse Autoencoders (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of refusal in instruction-tuned language models identifies latent features that causally mediate refusal behaviors. |
| Approach: | They conduct a mechanistic study of refusal in instruction-tuned LLMs using sparse autoencoders . they identify latent features that causally mediate refusal behaviors using sparsed autoencoding . |
| Outcome: | The proposed method validates refusal-related features across multiple datasets. |
Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on evaluating large language models' ability to handle disagreement cases. |
| Approach: | They evaluate the performance of large language models in detecting offensive language at varying levels of agreement. |
| Outcome: | The proposed model improves detection accuracy and model alignment with human judgment by using disagreement samples in training. |
MMAC: A Multilingual, Multimodal Alignment Framework for Cultural Grounding Evaluation (2026.acl-long)
Copied to clipboard
Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu, Xiaoxue Gao, Bryan Chen Zhengyu Tan, Bowei Zou, Chang Liu, Yujia Hu, Xing Xie, Xiaoyuan Yi, Jing Yao, Chaojun Wang, Long Li, Rui Liu, Huiyao Liu, Koji Inoue, Ryuichi Sumida, Tatsuya Kawahara, Fan Xu, Lingyu Ye, Wei Tian, Dongjun Kim, Jimin Jung, Jaehyung Seo, Nadya Yuki Wangsajaya, Pham Minh Duc, Ojasva Saxena, Palash Nandi, Xiyan Tao, Wiwik Karlina, Tuan Luong, Keertana Arun Vasan, Roy Ka-Wei Lee, Nancy F. Chen
| Challenge: | Existing models lack cultural alignment across modalities and languages . a new framework to assess cultural awareness across linguistics and languages is needed . |
| Approach: | They propose a framework that integrates tri-modally aligned cultural benchmarks and a five-dimensional evaluation protocol to assess cross-country awareness disparities. |
| Outcome: | The proposed framework assesses cultural awareness disparities across modalities and languages . it is the first dataset aligned at the input level across text, image, and speech . |
Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | PixelHumor is a benchmark dataset of 2,800 annotated multi-panel comics designed to evaluate LMMs’ ability to interpret multimodal humor and recognize narrative sequences. |
| Approach: | PixelHumor is a benchmark dataset of 2,800 annotated multi-panel comics designed to evaluate LMMs’ ability to interpret multimodal humor and recognize narrative sequences. |
| Outcome: | Experiments with state-of-the-art LMMs reveal that top models achieve only 61% accuracy in panel sequencing, far below human performance. |
CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual Large Language Models (MLLMs) exhibit strong generalization across languages, yet they remain prone to hallucinations due to training data imbalances. |
| Approach: | They propose a cross-lingual Chain-of-Thought framework that enhances cross-linguistic alignment . the framework guides the model to reason in a high-resource language before generating answers in low-resourced language. |
| Outcome: | The proposed framework reduces hallucination rates by up to 62% and significantly improves factual knowledge transfer across language pairs. |
Prompting for Multimodal Hateful Meme Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Hateful meme classification requires complex reasoning and contextual background knowledge. |
| Approach: | They propose a simple yet effective prompt-based model that prompts pre-trained language models for hateful meme classification. |
| Outcome: | The proposed model outperforms state-of-the-art models on hateful meme classification task. |
HateXScore: A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks do not assess why a text is deemed hateful . authors present a new metric to evaluate the reasoning quality of model explanations . |
| Approach: | They propose a metric suite to evaluate the reasoning quality of model explanations. |
| Outcome: | The proposed metric validates it as a practical tool for trustworthy and transparent moderation on six diverse hate speech datasets. |
Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can struggle to balance gullibility to misinformation and resistance to valid corrections in persuasive dialogues. |
| Approach: | They propose a framework evaluating multi-turn stance-change dynamics across dual dimensions: persuasion type and domain. |
| Outcome: | The proposed framework improves LLM-3.1-8B-Instruct accuracy under misleading persuasion in safety contexts from 4.21% to 76.54%. |
SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Long-form text generation remains a challenge for large language models . generating extended sequences often leads to degraded coherence and logical consistency . |
| Approach: | They propose a framework that integrates explicit structured thinking into long-form text generation. |
| Outcome: | The proposed framework surpasses even larger-scale models in evaluation and human evaluation. |
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently been shown to deliver impressive performance in various NLP tasks. |
| Approach: | They propose a plan-and-solve (PS) prompting that includes a few manual steps to generate reasoning steps and improves the quality of generated reasoning steps. |
| Outcome: | The proposed strategy outperforms Zero-shot-CoT on ten reasoning problems and has comparable performance to 8-shot CoT prompting on the math reasoning problem. |
LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderators (2025.emnlp-demos)
Copied to clipboard
| Challenge: | a new study shows that moderation systems that ignore localisation and low-resource variants risk degraded performance and exploitation in real-world deployments. |
| Approach: | They propose a lightweight, multilingual moderation classifier tailored to Singapore's context . it uses pre-trained OpenAI embeddings and a multi-head ordinal classifier . |
| Outcome: | The proposed classifier outperforms commercial and open-source models across 17 benchmarks. |
Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore’s Low-Resource Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformed natural language processing, but their safety mechanisms remain under-explored in low-resource, multilingual settings. |
| Approach: | They propose a red-teaming approach to probe LLM vulnerabilities in Singapore's diverse linguistic context using a dataset and evaluation framework. |
| Outcome: | The proposed framework systematically probes LLM vulnerabilities in three real-world scenarios including Singlish, Chinese, Malay, and Tamil. |
On Orthogonality Constraints for Transformers (2021.acl-short)
Copied to clipboard
Aston Zhang, Alvin Chan, Yi Tay, Jie Fu, Shuohang Wang, Shuai Zhang, Huajie Shao, Shuochao Yao, Roy Ka-Wei Lee
| Challenge: | a dedicated study on orthogonality constraints for transformers has been lacking . plug-and-play constraints increase the BLEU of transformers . |
| Approach: | They propose to use plug-and-play constraints to encourage matrices to be orthogonal for numerical stability. |
| Outcome: | The proposed constraint increases the BLEU on the large-scale WMT’16 EnDe benchmark by a factor of 28.4 to 29.6. |
BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models (2026.eacl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
| Challenge: | Existing evaluations assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. |
| Approach: | They propose a multimodal, multicultural benchmark to evaluate the robustness of everyday cultural knowledge in vision-language models across linguistic rephrasings and visual modalities. |
| Outcome: | ‘BLEnD-Vis‘ constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats. |
Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in simulating human behaviour and social intelligence, but they risk perpetuating societal biases, especially when demographic information is involved. |
| Approach: | They propose a framework that measures semantic shifts in responses and an LLM-judged Preference Win Rate to assess how demographic prompts affect response quality across power-disparate social scenarios. |
| Outcome: | The proposed framework measures semantic shifts in responses and an LLM-judged Preference Win Rate (WR) to assess how demographic prompts affect response quality across power-disparate social scenarios. |
Lost in Stories: Consistency Bugs in Long Story Generation by LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors unexplored. |
| Approach: | They propose a benchmark to evaluate narrative consistency in long-form story generation. |
| Outcome: | Evaluating LLMs, we find consistency errors are common in factual and temporal dimensions . authors say the findings can inform future efforts to improve consistency in long-form narrative generation. |
Graph-to-Tree Learning for Solving Math Word Problems (2020.acl-main)
Copied to clipboard
| Challenge: | Existing tree-based neural models do not capture the relationships and order information among the quantities well. |
| Approach: | They propose a novel deep learning architecture that combines the merits of the graph-based encoder and tree-based decoder to generate better solution expressions. |
| Outcome: | The proposed framework outperforms the state-of-the-art on two available datasets significantly. |
Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models generate unintended outputs due to their unsupervised nature. |
| Approach: | They propose a method to construct preference pairs of selected and rejected LLMs by repeated random sampling to improve alignment performance. |
| Outcome: | The proposed method improves performance as the sample size increases. |
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty (2026.findings-acl)
Copied to clipboard
| Challenge: | incorporating difficult prompts into training fails to enhance overall performance, e.g., as prompt difficulty decreases. |
| Approach: | They investigate how prompts of varying difficulty influence self-play preference optimization . they use the reward of sampled responses of a prompt as a proxy for its difficulty . |
| Outcome: | The proposed model improves on difficult prompts and easy prompts, but fails to train on difficult ones and learns from failures. |
Improving Text Auto-Completion with Next Phrase Prediction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Language models such as GPT-2 require considerable training effort to adapt to specific writing domains (e.g., medical). |
| Approach: | They propose an intermediate training strategy that encourages language models to complete partial queries with enriched phrases and eventually improve their text auto-completion performance. |
| Outcome: | The proposed approach outperforms baselines in auto-completion tasks for email and academic-writing domains with only around 1.2B tokens. |
HateGAN: Adversarial Generative-Based Data Augmentation for Hate Speech Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to detect online hate speech depend heavily on labeled datasets for training, which results in poor detection performance of the hate speech class. |
| Approach: | They propose a deep generative reinforcement learning model which augments two commonly-used hate speech detection datasets with the HateGAN generated tweets. |
| Outcome: | The proposed model improves the detection performance of hate speech class regardless of the classifiers and datasets used in the detection task. |